Asymptotic distribution and sparsistency for `1 penalized parametric M-estimators, with applications to linear SVM and logistic regression

Since its early use in least squares regression problems, the `1-penalization framework for variable selection has been employed in conjunction with a wide range of loss functions encompassing regression, classification and survival analysis. While a well developed theory exists for the `1-penalized least squares estimates, few results concern the behavior of `1-penalized estimates for general loss functions. In this paper, we derive two results concerning penalized estimates for a wide array of penalty and loss functions. Our first result characterizes the asymptotic distribution of penalized parametric Mestimators under mild conditions on the loss and penalty functions in the classical setting (fixed-p-largen). Our second result explicits necessary and sufficient generalized irrepresentability (GI) conditions for `1-penalized parametric M-estimates to consistently select the components of a model (sparsistency) as well as their sign (sign consistency). In general, the GI conditions depend on the Hessian of the risk function at the true value of the unknown parameter. Under Gaussian predictors, we obtain a set of conditions under which the GI conditions can be re-expressed solely in terms of the second moment of the predictors. We apply our theory to contrast `1-penalized SVM and logistic regression classifiers and find conditions under which they have the same behavior in terms of their model selection consistency (sparsistency and sign consistency). Finally, we provide simulation evidence for the theory based on these classification examples. ∗Indiana University, [email protected], [email protected] †Renmin University of China, [email protected] ‡University of California, Berkeley, [email protected]